Image processing device configured to correct output data based upon a point spread function

ABSTRACT

An image-capturing device includes: a plurality of microlenses arranged in a two-dimensional manner; a plurality of photodetectors arranged to correspond to the plurality of microlenses; an image synthesizing part that synthesizes an image at an optional image plane of a photographic optical system, based on signals from the plurality of photodetectors; and an image processing part that removes signals of other pixels from signals of pixels constituting the image synthesized in the image synthesizing part.

TECHNICAL FIELD

The present invention relates to an image-capturing device.

BACKGROUND ART

Conventionally, an image-capturing device synthesizing an image focusedon an optional image plane from data captured in one photographic shothas been known. For example, an image-capturing device described inpatent literature 1 synthesizes image data based on output values ofpixels receiving rays passing through a photographic optical system andbeing incident on the centers of a plurality of microlenses.

CITATION LIST Patent Literature

PATENT LITERATURE 1: Japanese Laid-Open Patent Publication No. 2007-4471

SUMMARY OF INVENTION Technical Problem

In the prior art, there is a problem of decreasing a contrast ofsynthetic image data, when a width of a virtual pixel is set narrowerthan a pitch of a microlens in order to increase a resolution of thesynthetic image data.

Solution to Problem

According to the 1st aspect of the present invention, an image-capturingdevice comprises: a plurality of microlenses arranged in atwo-dimensional manner; a plurality of photodetectors arranged tocorrespond to the plurality of microlenses; an image synthesizing partthat synthesizes an image at an optional image plane of a photographicoptical system, based on signals from the plurality of photodetectors;and an image processing part that removes signals of other pixels fromsignals of pixels constituting the image synthesized in the imagesynthesizing part.

According to the 2nd aspect of the present invention, in theimage-capturing device according to the 1st aspect, it is preferredthat: signals from a plurality of photodetectors corresponding to thepixel and signals from a plurality of photodetectors corresponding tothe other pixels include signals from overlapping photodetectors; andthe image processing part removes signals of the overlapping otherpixels from the signals from the plurality of photodetectorscorresponding to the pixel.

According to the 3rd aspect of the present invention, an image-capturingdevice comprises: a plurality of microlenses arranged in atwo-dimensional manner so that a light flux passing through aphotographic optical system is incident thereon; a plurality ofphotodetectors arranged behind the plurality of microlenses tocorrespond to each of the plurality of microlenses; an imagesynthesizing part that synthesizes image data of an image at an optionalimage plane of the photographic optical system, based on outputs of theplurality of photodetectors corresponding to each of the plurality ofmicrolenses; a Fourier transforming part that performs a Fouriertransform on the image data synthesized in the image synthesizing part;an operating unit that effectively divides a result of the Fouriertransform in the Fourier transforming part, by a Fourier image of apoint spread function representing an optical divergence of the lightflux incident on the plurality of microlenses; and an inverse Fouriertransforming part that performs an inverse Fourier transform on a resultof the division in the operating unit to create target image data.

According to the 4th aspect of the present invention, in theimage-capturing device according to the 3rd aspect, it is preferred thatthe operating unit effectively divides the result of the Fouriertransform in the Fourier transforming part, by applying a Wiener filterbased on the Fourier image of the point spread function to the result ofthe Fourier transform in the Fourier transforming part.

According to the 5th aspect of the present invention, in theimage-capturing device according to the 3rd or 4th aspect, it ispreferred that a size of the point spread function is equal to a size ofa region covered by one microlense.

According to the 6th aspect of the present invention, in theimage-capturing device according to the 5th aspect, it is preferred thatthe number of elements of the point spread function is equal to thenumber of the plurality of photodetectors covered by one microlens.

According to the 7th aspect of the present invention, in theimage-capturing device according to any one of the 3rd through 6thaspects, it is preferred that the point spread function is determinedfrom an arrangement of the plurality of microlenses and an arrangementof pixels synthesized in the image synthesizing part.

Advantageous Effect of Invention

According to the present invention, it is possible to synthesize imagedata having both a high contrast and a high resolution.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A view showing an configuration of a digital camera according toa first embodiment.

FIG. 2 A perspective view of an image-capturing unit 100.

FIG. 3 A cross-sectional view schematically showing a photographic lensL1 and an image-capturing unit 100.

FIG. 4 A cross-sectional view schematically showing a light flux from alight point on an image plane to be synthesized there, and theimage-capturing unit 100.

FIG. 5 A schematic view showing divergence of lights from light pointsP4 and P5 on respective different image planes.

FIG. 6 A plan view of a part of the image-capturing plane of the imagesensor 13 that is covered by 25 microlenses 120 shown in FIG. 5(b), asseen in the optical axis direction.

FIG. 7 A view for explaining a method for determining each incidentregion on an image-capturing plane in a case where a light flux from onelight point is incident on a plurality of microlenses 120.

FIG. 8 A view showing a light point P7 shifted to the left by a distancep from the position of the light point P5, in the image plane S5 shownin FIG. 5.

FIG. 9 A schematic view showing how light fluxes from light pointsoverlap each other.

FIG. 10 A view showing an example of an output distribution of syntheticpixels.

FIG. 11 A view showing an example of a PSF.

FIG. 12 A plan view of a microlens array 12 according to the secondembodiment, as seen in the optical axis direction.

FIG. 13 A conceptual view for explaining how a contrast to be decreasedis compensated by applying a high pass filter.

FIG. 14 A view showing a configuration of a control circuit 101 A forimplementing the forth variation.

DESCRIPTION OF EMBODIMENTS

(First Embodiment)

A digital camera according to this embodiment creates image data havinga focal position desired by the user, with numerical processing, byutilizing the fact that an image signal acquired by photography througha microlens array has wavefront information such as depth information. Alight flux (bundle of rays) from a subject, which is incident through aphotographic lens, converges in the vicinity of the microlens array. Inthis regard, a position at which the light flux converges varies in anoptical axis direction of the photographic lens, depending on theposition of the subject. Furthermore, light fluxes from the subject donot converge on one and the same plane if the subject is athree-dimensional object. The digital camera according to thisembodiment creates (synthesizes) image data that reproduces a subjectimage formed at an imaging position in the optical axis directiondesired by the user. The created image data will be hereinafter referredto as synthetic image data. The synthetic image data appears as if afocal point of an imaging optical system were at this imaging position(not a real imaging position, but the position desired by the user).Thus, in the following description, this imaging position will bereferred to as a focal position.

Furthermore, the digital camera according to this embodiment isconfigured to be able to create synthetic image data having a resolutionthat is larger than the number of microlenses included in the microlensarray. The digital camera includes a large number of microlenses and aplurality of image-capturing pixels (photodetectors (light receivingelements)) corresponding to each individual microlens. In order tocreate synthetic image data having a focal position selected by theuser, the digital camera uses not only image signals output fromimage-capturing pixels corresponding to one microlens, but also imagesignals output from image-capturing pixels corresponding to microlensesarranged around said microlens to create a synthetic image signal whichcorresponds to an imaging region for one pixel of the synthetic imagedata, thereby creating synthetic image data having a variable focalposition. This will be described in detail hereinafter.

FIG. 1 is a view showing a configuration of the digital camera accordingto the first embodiment. The digital camera 1 is configured such that aninterchangeable lens 2 having a photographic lens L1 is removablyattached thereto with a bayonet-type lens mount mechanism, for example.The digital camera 1 includes an image-capturing unit 100, a controlcircuit 101, an A/D conversion circuit 102, a memory 103, an operatingunit 112, a display device 109, a LCD drive circuit 110, and a memorycard interface 111. The image-capturing unit 100 includes a microlensarray 12 having a large number of microlenses 120 arranged in atwo-dimensional manner, and an image sensor 13. It should be noted thatin the following description, the Z-axis is set to be parallel to anoptical axis of the photographic lens L1, while the X-axis and theY-axis are set to be orthogonal to each other in a plane orthogonal tothe Z-axis.

The photographic lens L1 is composed of a plurality of groups of opticallenses and converges a light flux from a subject in the vicinity of afocal plane of the photographic lens L1. The photographic lens L1 isprovided with an aperture 11 for adjusting an incident light amount. Forthe purpose of explanation, the photographic lens L1 is represented byone representative lens in FIG. 1. In the vicinity of a predeterminedimaging plane of the photographic lens L1, the microlens array 12 andthe image sensor 13 are sequentially arranged. The image sensor 13 iscomposed of a CCD or a CMOS image sensor having a plurality ofphotoelectric conversion elements. The image sensor 13 captures asubject image formed on the image-capturing plane and outputs aphotoelectric conversion signal (an image signal), which is dependent onthe subject image, to the A/D conversion circuit 102 under the controlof the control circuit 101. The image-capturing unit 100 will bedescribed in detail hereinafter.

The A/D conversion circuit 102 is a circuit processing the image signaloutput from the image sensor 13 in an analogous manner and thenconverting the image signal to a digital image signal. The controlcircuit 101 is composed of a CPU and peripheral circuits such as amemory or the like. The control circuit 101 reads and executes a controlprogram previously stored in a ROM (not shown) or the like. This controlprogram causes the control circuit 101 to perform a predeterminedcalculation by using signals input from parts constituting the digitalcamera 1 and send out control signals to the parts of the digital camera1, in order to control photographic operations. Moreover, the controlcircuit 101 determines a focal position of synthetic image data, basedon an operating signal input from the operating unit 112 in response tooperation of a focal position input button 112 a, as describedhereinafter.

The control circuit 101 functionally includes an image synthesizing part105, a Fourier transforming part 106, a dividing part 107, and aninverse Fourier transforming part 108. The image synthesizing part 105synthesizes synthetic image data at an optionally selected focal planedifferent from the predetermined focal plane of the photographic lensL1. The Fourier transforming part 106 performs a Fourier transform onthe synthetic image data synthesized in the image synthesizing part 105.The dividing part 107 effectively divides the result of the Fouriertransform by a Fourier image of a point spread function describedhereinafter. The inverse Fourier transforming part 108 performs aninverse Fourier transform on the result of the division in the dividingpart 107 to create target image data. The image synthesizing part 105,the Fourier transforming part 106, the dividing part 107, and theinverse Fourier transforming part 108 will be described in detailhereinafter.

The memory 103 is a volatile storage media used for temporarily storingthe image signal that has been digitally converted by the A/D conversioncircuit 102, or data during or after image processing, image compressionprocessing, and display image data creation processing. The memory cardinterface 111 is an interface to which a memory card 111 a is removablyattached. The memory card interface 111 is an interface circuit forwriting image data into the memory card 111 a or reading image datarecorded in the memory card 111 a, under the control of the controlcircuit 101. The memory card 111 a is a semiconductor memory card, suchas a Compact Flash (registered trademark) or a SD card.

The LCD drive circuit 110 is a circuit for driving the display device109 based on instructions of the control circuit 101. The display device109 is composed of a liquid crystal panel or the like and displaysdisplay data created in the control circuit 101 based on the image datarecorded in the memory card 111 a, in a reproducing mode. A menu screenfor setting various operations of the digital camera 1 is also displayedon the display device 109.

The operating unit 112 accepts operations from the user and outputsvarious operating signals, which are dependent on contents of theoperations, to the control circuit 101. The operating unit 112 includesa focal position input button 112 a, a power button, a release button, adisplay switching button for other setting menus, and a setting menuselect button, for example. The focal position input button 112 a isoperated by the user to input a focal position y of the synthetic imagedata. Once the focal position y is selected by the operation of thefocal position input button 112 a by the user, the operating unit 112outputs an operating signal including the focal position y to thecontrol circuit 101.

A configuration of the image-capturing unit 100 will now be described indetail, with reference to a perspective view of the image-capturing unit100 shown in FIG. 2. The image-capturing unit 100 has the microlensarray 12 and the image sensor 13. The microlens array 12 has a pluralityof microlenses 120 which are squarely arranged in a two-dimensionalmanner on the XY plane. In the image sensor 13, photoelectric conversionelements 130 (hereinafter referred to as image-capturing pixels 130) arearranged in a two-dimensional manner, in an arrangement patterncorresponding to the microlenses 120. The photoelectric conversionelements 130 receive light passing through each microlens 120. The imagesensor 13 is arranged at a distance of a focal length f of the microlens120, from the microlens array 12. In other words, for each microlens120, a plurality of image-capturing pixels 13 corresponding to (orcovered by) the microlens 120 are provided at a distance of the focallength f of the microlens 120.

It should be noted that only a part of the plurality of microlenses 120provided in the microlens array 12 and the plurality of image-capturingpixels 130 provided in the image sensor 13 is shown in FIG. 2. Inpractice, there are more microlenses 120 and image-capturing pixels 130.For example, approximately 100 image-capturing pixels 130 are covered byone microlens 120 and the number of the microlenses 120 included in themicrolens array 12 is thus approximately 1/100 of the number of theimage-capturing pixels 130 included in the image sensor 13.

For example, given that the focal length of the photographic lens L1 is50 millimeter, a position of a so-called exit pupil of the photographiclens L1 can be then considered to be virtually at infinity compared withthe microlens 120, as seen from the image sensor 13, because the focallength f of the microlens 120 is approximately several hundredsmicrometer (approximately 1/100 of the focal length of the photographiclens L1). Thus, the position of the exit pupil of the photographic lensL1 and the image-capturing plane of the image sensor 13 can beconsidered to be optically conjugate to each other.

FIG. 3 is a cross-sectional view schematically showing the photographiclens L1 and the image-capturing unit 100. The microlens array 12 isprovided in the vicinity of the predetermined imaging plane S1 of thephotographic lens L1 shown in the left side in FIG. 3, and the imagesensor 13 is provided in the vicinity of the focal position S2 of themicrolens 120.

The subject image formed in the vicinity of the microlens array 12 bythe photographic lens L1 is compressed by each microlens 120 included inthe microlens array 12 and convoluted in the image sensor 13. Forexample, if an image magnification of the photographic lens L1 is 1/50,or if the photographic lens L1 forms a subject image that is 1/50 thesize of the real subject on the predetermined imaging plane S1, thesubject image is formed with the magnification of 1/2500, the square of1/50, as seen in the depth direction. Thus, the photographic lens L1forms a stereoscopic image, in which the subject in a three-dimensionalspace is compressed in the depth direction, on the predetermined imagingplane S1.

FIG. 4 is a cross-sectional view schematically showing a light flux froma light point on an image plane to be synthesized there, and theimage-capturing unit 100. In FIG. 4, a light point P1 provided on animage plane S3 to be synthesized there will be considered. A divergenceangle θ1 of a light directing from the light point P1 to theimage-capturing unit 100 is restricted by a pupil size of thephotographic lens L1 (i.e. a F-number of the photographic lens L1). Ifthe F-number of the microlens 120 is equal to or smaller than theF-number of the photographic lens L1, the light flux emitting from thelight point P1 and being incident on a microlens 120 does not divergeout of a region covered by the microlens 120.

Here, if the light flux from the light point P1 is incident on fivemicrolenses 120 a-120 e as shown in FIG. 4, a total amount of radiationfrom the light point P1 restricted by the pupil can be obtained byintegrating incident light amounts on the image-capturing planes oflight fluxes 30 a-30 e incident on the microlenses 120 a-120 e(photodetection outputs of a group of image-capturing pixels 130 a-130e).

In summary, in order to create a certain synthetic pixel constitutingsynthetic image data, a total light amount of a light cross-section onthe image-capturing plane corresponding to coordinates of the syntheticpixel is to be calculated. For the purpose of performing thiscalculation, the image synthesizing part 105 first (1) determinesmicrolenses 120 on which a light flux from a certain light point on animage plane to be synthesized there is incident, and then (2) determinesimage-capturing pixels on which the light flux from the light point isincident, for each determined microlens 120.

(1) In order to determine microlenses 120 on which a light flux from acertain light point on an image plane to be synthesized there isincident, it is only necessary to determine how the light flux from thelight point diverges. As described above, the divergence angle of thelight from the light point can be specified by the pupil of thephotographic lens L1. It is assumed in the following description thatthe F-number of the photographic lens L1 is equal to the F-number of themicrolens 120.

FIG. 5(a) is a cross-sectional view schematically showing divergence oflights from light points P4 and P5 on respective different image planes,and FIG. 5(b) is a plan view thereof, as seen from the side of thephotographic lens L1. A light flux from the light point P4 located on animage plane S4 spaced apart from the microlens 120 with a distance ofthe focal length f of the microlens 120 is incident on the microlensarray 12, while diverging to an extent 31 of exactly one microlens 120.

On the other hand, a light flux from the light point P5 located on animage plane S5 spaced apart from the microlens 120 with a distance oftwo times the focal length f of the microlens 120 (i.e. 2f) diverges toan extent 32 larger than one microlens 120, as shown in both FIG. 5(a)and FIG. 5(b).

In this way, microlenses 120 on which a light flux from a certain lightpoint is incident can be determined based on the distance from the lightpoint to the microlens 120 (i.e. the distance from the image plane to besynthesized there to the microlens 120). It should be noted that inpractice it is necessary to determine these microlenses 120 also inconsideration of the F-number of the photographic lens L1. For example,if the F-number of the photographic lens L1 is larger than the F-numberof the microlens 120 (the photographic lens L1 is darker than themicrolens 120), the divergence of the light flux from the light point issmaller.

The image synthesizing part 105 then (2) determines, for each determinedmicrolens 120, image-capturing pixels on which the light flux from thelight point is incident. If the light point is located at a distance ofthe focal length f from the microlens 120, such as the case of the lightpoint P4 in FIG. 5, a circular light extends over the overall regioncovered by the microlens 120 directly under the light point P4.Accordingly, all image-capturing pixels 130 covered by one microlens 120are to be selected in this case. If the light point is located at adistance less than the focal length f from the microlens 120, divergenceof the light remains within the region covered by one microlens 120because of restricted divergence angle of the incident light flux, eventhough the light does not converge, but diverges in the microlens 120.

On the other hand, if the light point is located at a distance largerthan the focal length f, the light flux from the light point is incidenton a plurality of microlenses 120. Therefore, it is required to selectimage-capturing pixels 130 needed for creation of a synthetic pixel,among a large number of image-capturing pixels 130 covered by theplurality of microlenses 120. In the following, a case where the lightpoint is located at a distance larger than the focal length f will bedescribed with reference to an example of the light point P5 in FIG. 5,i.e. a case where the light point is located at a distance of 2f fromthe microlens 120.

If the light point is located at a distance of 2f, the light flux fromthe light point is incident on a total of nine microlenses, i.e. amicrolens 120 directly under the light point and eight adjacentmicrolenses 120, as shown in FIG. 5(b).

FIG. 6 is a plan view of a part of the image-capturing plane of theimage sensor 13 that is covered by 25 microlenses 120 shown in FIG.5(b), as seen in the optical axis direction. In FIG. 6, a position ofeach microlens 120 in FIG. 5(b) is shown by a dashed line, overlayingthe image-capturing plane of the image sensor 13.

The light flux from the light point P5 shown in FIG. 5(a) is incident onnine regions 33 a-33 i shown in FIG. 6. As apparent from FIG. 6, theregions 33 a-33 i have shapes obtained by segmenting or dividing theregion covered by one microlens 120 into nine segments. In this way, thesize of the region on the image-capturing plane where a light flux froma light point is incident (the size of the light cross-section) isalways same as the size of the region covered by one microlens 120,irrespective of the position of the light point.

A method for determining each incident region on an image-capturingplane in a case where a light flux from one light point is incident on aplurality of microlenses 120 as in FIG. 6 will now be described withreference to FIG. 7. FIG. 7(a) is a plan view of a part of the microlensarray 12, as seen in the optical axis direction. The control circuit 101first determines a range over which the light flux from the light pointP6 shown in the center of FIG. 7(a) extends. In this example, a region34 spanning nine microlenses 120 a-120 i is the range over which thelight flux from the light point P6 extends.

Then, as shown in FIG. 7(b), the region 34 is segmented by a grid havinga pitch corresponding to each microlens 120. In FIG. 7(b), the region 34over which the light flux from the light point P6 extends is segmentedby square grid segments 35 a-35 i having a size equivalent to thediameter of each microlens 120, each grid segment being arranged tocorrespond to respective one of nine microlenses 120 a-120 i shown inFIG. 7(a).

Finally, each divided region 34 a-34 i is arranged in a region coveredby its respective corresponding microlens 120 a-120 i, as shown in FIG.7(c). Relative positions of the divided regions 34 a-34 i in therespective covered regions are identical to relative positions of ninesquare segments 35 a-35 i shown in FIG. 7(b). For example, the dividedregion 34 a corresponding to the square segment 35 a at the top leftcorner, among the square segments 35 a-35 i arranged in a 3×3 matrix, isarranged at the top left corner of the corresponding covered region 36a.

In this way, given that the height of the light point is h, the focallength of the microlens 120 is f, and the diameter of the microlens 120is d, the image-capturing pixels on which the light flux from the lightpoint is incident can be determined by segmenting the region over whichthe light flux from the light point extends by a grid having a width ofd/(h/f). It should be noted that (h/f) can be a negative value. In sucha case, the light point is considered to be located on the image sensor13 side of the microlens 120.

Although the method for creating a synthetic pixel has been describedwith reference to an example in which a light point is located on a lenscenter axis of a certain microlens 120, this method for creating thesynthetic pixel is also applicable to a case where the light point islocated at a position other than that. If only the light points on thelens center axis of the certain microlens 120 were synthesized, thenumber of pixels of synthetic image data to be created would be limitedto the number of the microlenses 120. As a result, only synthetic imagedata having a small number of pixels could be created. Hence, the imagesynthesizing part 105 in this embodiment sets a larger number of lightpoints on the image plane to be synthesized there and creates syntheticpixels for these light points with the above-described synthesizingmethod, in order to create synthetic image data having a larger numberof pixels.

FIG. 8 is a view showing a light point P7 shifted to the left by adistance p from the position of the light point P5, in the image planeS5 shown in FIG. 5. In this case, a range 37 over which the light fluxextends is also shifted to the left by the distance p. A synthetic pixelcorresponding to the light point P7 can be therefore created by settingdivided regions for the shifted range 37, in the same manner as in FIG.7. For example, for one microlens 120, each light point is set at aposition shifted by d/4 from its adjacent light point in both verticaland horizontal directions and a synthetic pixel is created for eachlight point. Consequently, sixteen synthetic pixels can be created fromone microlens 120. In this case, the synthetic image data has the numberof pixels that is sixteen times the number of the microlenses 120.

If a plurality of light points are provided on one microlens 120 toperform an image synthesis as described above, an output of oneimage-capturing pixel 130 is used for creation of a plurality ofsynthetic pixels. In practice, a large number of light points areconsecutively located in a three-dimensional manner on an image plane tobe synthesized there and lights from these light points are overlappedon the image-capturing plane of the image sensor 13. Therefore, for agiven light point provided on the microlens 12, a photodetection outputof the image-capturing pixel 130 on which a light flux from the lightpoint is incident includes outputs corresponding to light fluxes fromlight points other than said light point.

FIG. 9 is a schematic view showing how light fluxes from light pointsoverlap each other. FIG. 9(a) is a cross-sectional view of the imageplane S4 to be synthesized there, the microlens array 12, and the imagesensor 13, taken along the Y-Z plane, and FIG. 9(b) is a plan view ofthe image-capturing plane of the image sensor 13, as seen in the opticalaxis direction. Five light points P8-P12 arranged in a line on the imageplane S4 spaced apart from the microlens 120 with a distance of thefocal length f of the microlens 120 will be considered below.

The five light points P8-P12 are provided at intervals of one-half of awidth d of the microlens 120 (i.e. d/2). Specifically, the light pointsP8, P10, P12 are set on the center axes of their respective differentmicrolenses 120, while the light points P9, P11 are set between twoadjacent microlenses 120. In this case, light fluxes 40-44 from thelight points P8-P12 are incident on regions 45-49 of the image-capturingplane of the image sensor 13, while overlapping each other.

For example, the light flux 41 from the light point P9 is incident onthe region 46 shown in FIG. 9(b) and this region 46 overlaps the region45 on which the light flux 40 from the light point P8 is incident andthe region 47 on which the light flux 42 from the light point P10 isincident. In other words, a photodetection output from animage-capturing pixel 130 corresponding to an overlap region between theregion 45 and the region 46 is a result of overlapping an outputcorresponding to the light flux 40 from the light point P8 and an outputcorresponding to the light flux 41 from the light point P9.

Here, given that a true strength of an i-th light point is ai, an outputPi of a synthetic pixel corresponding to the i-th light point can becalculated with the following equation (1).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\{P_{i} = {\frac{a_{i - 1} + a_{i + 1}}{2} + a_{i}}} & (1)\end{matrix}$

The control circuit 101 segments or divides one microlens 120 into fourin both vertical and horizontal directions to provide sixteen lightpoints. In this case, the output Pi of the synthetic pixel can belikewise calculated with the following equation (2) for four lightpoints arranged alongside each other.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{P_{i} = {\frac{a_{i - 3} + a_{i + 3}}{4} + \frac{2 \cdot \left( {a_{i - 2} + a_{i + 2}} \right)}{4} + \frac{3 \cdot \left( {a_{i - 1} + a_{i + 1}} \right)}{4} + a_{i}}} & (2)\end{matrix}$

An output distribution of the synthetic pixel in this case is shown inFIG. 10. An ordinate in FIG. 10 is pixel output and the abscissa ispixel position. As apparent from FIG. 10, the synthetic pixel is a pixeloutput obtained from a convolution integral of a triangle. A pointspread function (PSF), which represents how the light flux from thelight point diverges under the effect of the microlens 120, can bedetermined from such a output distribution.

The PSF is determined from an arrangement of a plurality of microlenses120 and an arrangement of pixels to be synthesized in the imagesynthesizing part 105. In this embodiment, it is assumed that the sizeof the PSF is equal to the size of the region covered by one microlens120. Additionally, it is assumed that the number of elements of the PSFis equal to the number of a plurality of image-capturing pixels 130covered by one microlens 120.

Although the output distribution of the synthetic pixel is shown in onedimension in FIG. 10, it is necessary to calculate the distribution intwo dimensions in order to determine the PSF. In other words, the PSFcan be determined by performing the calculation in consideration of notonly overlapping of light fluxes from right and left light points, butalso overlapping of light fluxes from all surrounding light points.

FIG. 11(a) is a view showing values of the derived PSF and FIG. 11(b) isa view in which the PSF is plotted in three-dimensional coordinates. InFIG. 11, the values of the PSF are shown for 81 points in one microlens120.

The dividing part 107 uses the PSF derived as described above in orderto decompose the overlapping of the image outputs shown in FIG. 10. Theoverlapping of the image outputs can be represented by the followingequation (3). For the sake of simplicity, the equation (3) is hereexpressed in one dimension.

[Math. 3]i(x)=∫_(−∞) ^(+∞) psf(t)·f(x−t)dt  (3)

In the above equation (3), psf(t) represents PSF, f(x) represents truelight strength, and i(x) represents overlapped image output. Becausepsf(t) is known from the above-described method and the overlapped imageoutput i(x) is also known, the true light strength f(x) can bedetermined. Then, a Fourier transform is performed on i(x) to representthe above equation (3) as a product of a Fourier transformed imagePSF(u) of the PSF and a Fourier transformed image F(u) of the true lightstrength f(x). A Fourier transformed image I(u) of i(x) is thusrepresented by the following equation (4).

[Math. 4]I(u)=PSF(u)*F(u)  (4)

From the above equation (4), the Fourier transformed image F(u) of thetrue light strength f(x) can be represented by the following equation(5).

[Math. 5]F(u)=I(u)/PSF(u)  (5)

Because the functions i(x) and psf(x) and their Fourier transformedimages I(u) and PSF(u) are already known, the Fourier transformed imageF(u) of the true light strength f(x) can be derived from the aboveequation (5). Thereafter, an inverse Fourier transform is performed onthe Fourier transformed image F(u) to determine the true light strengthf(x).

However, in practice, when the above-described calculation is performedto determine the true light strength f(x), a high-frequency noise causedby arithmetic errors appears in the overall synthetic image data, whichresults in an unclear image. For this reason, the dividing part 107 inthis embodiment uses a well-known Wiener filter to suppress thehigh-frequency noise. By applying the Wiener filter to the aboveequation (5), it becomes the following equation (6). The dividing part107 determines the Fourier transformed image F(u) of the true lightstrength f(x) with the following equation (6), instead of the aboveequation (5), to create a clear synthetic image data in which thehigh-frequency noise is suppressed. In the following equation (6),PSF*(u) represents a complex conjugate of PSF(u).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 6} \right\rbrack & \; \\{{F(u)} = {\frac{{PSF}^{*}(u)}{\left| {{PSF}(u)} \middle| {}_{2}{+ \delta} \right.} \cdot {I(u)}}} & (6)\end{matrix}$

The above-described sequence will be briefly described with reference toa block view shown in FIG. 1. An image-capturing signal output from theimage sensor 13 is converted to a digital image signal by the A/Dconversion circuit 102 and stored in the memory 103. A focal position tobe synthesized there and an aperture value determining a depth of fieldare input from the operating unit 112 to the control circuit 101. Theimage synthesizing part 105 synthesizes a two-dimensional image with theinput focal position and aperture value.

On the other hand, the Fourier transforming part 106 determines a pointspread function (PSF) for one light point from the arrangement of themicrolenses 120 in the microlens array 12 and the position of thesynthetic pixels from the image synthesizing part 105, and performs aFourier transform on the PSF and the synthetic image by the imagesynthesizing part 105. The dividing part 107 effectively divides theFourier image of the synthetic image by the image synthesizing part 105by the Fourier image of the PSF with the Wiener filter, as in the aboveequation (6), to obtain the Fourier image F(u) of the true lightstrength f(x). Finally, the inverse Fourier transforming part 108performs an inverse Fourier transform on the Fourier image F(u) toobtain target image data which is improved over the original syntheticimage data. This image is an image having a high contrast, withoverlapping of the incident light flux by the microlens 120 beingdecomposed.

The following operations and advantageous effects can be achieved withthe digital camera according to the above-described first embodiment.

(1) A digital camera 1 includes a microlens array 12 having a pluralityof microlenses 120 arranged in a two-dimensional manner so that a lightflux passing through the photographic lens L1 is incident thereon, andan image sensor 13 having a plurality of image-capturing pixels 130arranged behind the plurality of the microlenses 120 to correspond toeach of the plurality of the microlenses 120, respectively. A controlcircuit 101 includes an image synthesizing part 105 for synthesizingimage data of an image at an optional or any image plane of thephotographic lens L1, based on outputs of the plurality ofimage-capturing pixels 130 corresponding to respective one of theplurality of microlenses 120, a Fourier transforming part 106 forperforming a Fourier transform on the image data synthesized in theimage synthesizing part 105, a dividing part 107 for effectivelydividing the result of the Fourier transform by a Fourier image of apoint spread function representing an optical divergence of a light fluxincident on the plurality of microlenses 120, and an inverse Fouriertransforming part 108 for performing an inverse Fourier transform on theresult of the division in the dividing part 107 to create target imagedata. In this way, image data having both a high contrast and a highresolution can be synthesized.

(2) The dividing part 107 effectively divides the result of the Fouriertransform in the Fourier transforming part 106, by applying a Wienerfilter based on the Fourier image of the point spread function to theresult of the Fourier transform in the Fourier transforming part 106. Inthis way, a clear image in which the high-frequency noise is suppressedcan be obtained.

(Second Embodiment)

A digital camera according to a second embodiment of the presentinvention will now be described with reference to the drawings. The sameparts as in the first embodiment are denoted by the same referencecharacters as in the first embodiment and the description thereof willbe omitted herein.

FIG. 12 is a plan view of a microlens array 12 according to the secondembodiment, as seen in the optical axis direction. As shown in FIG. 12,hexagonal microlenses 120 forming a honeycomb structure are employed inthis embodiment. The present invention is also applicable to themicrolenses 120 having such a shape, in a similar manner to the firstembodiment.

A PSF used in this embodiment is different from that in the firstembodiment, because an arrangement of the microlenses 120 is differentfrom that in the first embodiment. A method for deriving the PSF is perse the same as that in the first embodiment. That is, overlapping oflight fluxes from surrounding light points such as in FIG. 10 is to becalculated in two dimensions.

The following variations are also within the scope of the presentinvention and one or more variations may be combined with theabove-described embodiments.

(First Variation)

Although the embodiments in which the present invention is applied to adigital cameras capable of having an interchangeable lens have beendescribed above, the present invention is not limited to theembodiments. The present invention is equally applicable to a camerahaving a built-in lens, for example.

(Second Variation)

The arrangement and shape of the microlenses 120 are not limited to thatdescribed in the first and second embodiments. This is also the case forthe arrangement and shape of each image-capturing pixel in theimage-capturing plane of the image sensor 13. For example,image-capturing pixels 130 covered by one microlens 120 may be separatefrom that covered by another microlens 120.

(Third Variation)

The true light strength f(x) may be determined without a Wiener filter.Thus, the dividing part 107 may be configured to determine F(u) with theabove equation (4), instead of the above equation (5).

(Forth Variation)

In the above-described embodiments, overlapping of incident light fluxescaused by the microlens 120 is decomposed by applying processings of theFourier transforming part 106, the dividing part 107, and the inverseFourier transforming part 108 to the synthetic image synthesized in theimage synthesizing part 105, in order to obtain an image having a highcontrast. However, a high pass filter may be applied to such a part ofan image before synthesis that has a high spatial frequency component,in order to previously compensate a contrast to be decreased.

It will now be described whether the high pass filter is effective tothe original image before synthesis, instead of the synthetic image.

For the sake of simplicity of the description, it is assumed that animage is in the vicinity of a focal position of a microlens and f(x) isa variation of strength as a function of one-dimensional position of theimage. The strength in the image sensor is f(x)/S, where S is a diameterof the microlens, because a light output diverges to an extent of S fromthis position. A position x₀ of a given image sensor plane is a positionwhere those lights are overlapped and therefore the following equationcan be obtained.

$\begin{matrix}{{I\left( x_{0} \right)} = {\int_{{- s}/2}^{s/2}{\frac{f\left( {x + x_{0}} \right)}{S}\ d\; x}}} & \left\lbrack {{Math}.\mspace{14mu} 7} \right\rbrack\end{matrix}$

The difference from an adjacent pixel corresponds to a derivative ofI(x₀) and therefore can be considered as:

$\begin{matrix}{{\frac{d}{d\; x}I\left( x_{0} \right)} = {\frac{1}{S}{\int_{{- s}/2}^{s/2}{\frac{d\;{f\left( {x + x_{0}} \right)}}{d\; x}\ d\; x}}}} & \left\lbrack {{Math}.\mspace{14mu} 8} \right\rbrack\end{matrix}$

The right side of this equation is exactly f(x). Therefore, the highpass filter which increases the difference value translates to anincrease in a gain of the original image f(x). Thus, a decrease incontrast of the display image is compensated by applying the high passfilter to the original image of the image sensor.

FIG. 13 is a conceptual view for explaining how a contrast to bedecreased is compensated by applying the high pass filter. If a spatialfrequency is lower than a density of microlenses, no problem arises withrespect to the contrast of the synthetic image. However, if the spatialfrequency is higher than the density of the microlenses, the contrast ofthe synthetic image is decreased due to the effect of the overlapping.For this reason, a part having a high spatial frequency is previouslyemphasized by performing high pass filter processing to the originalimage before synthesis. In this way, a decrease in contrast of the parthaving a high spatial frequency is compensated, when the image synthesisof an optional image plane is performed as described herein.

FIG. 14 is a view showing a configuration of a control circuit 101A forimplementing this variation. The control circuit 101A having a high passfilter part 201 and an image synthesizing part 105. The Fouriertransforming part 106, the dividing part 107, and the inverse Fouriertransforming part 108 of the control circuit 101 in FIG. 1 are omittedand the high pass filter part 201 is provided instead. The imagesynthesizing part 105 is identical to the image synthesizing part 105 inthe above-described embodiments and therefore denoted by the samereference character.

Once original image data before image synthesis is input from the memory103 to the control circuit 101A, the data is first input to the highpass filter part 201 and subjected to high pass filter processing.Thereafter, the data is input to the image synthesizing part 105 tosynthesize a two-dimensional image of a focal position and an aperturevalue instructed by the operating unit 112, in the same manner as in theabove-described embodiment. Thereafter, the synthetic image aftersynthesis is output.

In this way, a synthetic image in which a decrease in contrast iscompensated can be created, even with a simple configuration in whichhigh pass filter processing is applied to an original image beforesynthesis.

Unless impairing characteristics of the present invention, the presentinvention is not limited to the above-described embodiments, and, on thecontrary, other embodiments conceivable within the scope of thetechnical idea of the present invention are also encompassed within thescope of the present invention.

The disclosure of the following priority application is hereinincorporated by reference: Japanese Patent Application No. 2012-156940filed Jul. 12, 2012.

REFERENCE SIGNS LIST

1 . . . digital camera, 2 . . . interchangeable lens, 12 . . . microlensarray, 13 . . . image sensor, 101 . . . control circuit, 105 . . . imagesynthesizing part, 106 . . . Fourier transforming part, 107 . . .dividing part, 108 . . . inverse Fourier transforming part, 120 . . .microlens, 130 . . . image-capturing pixel, L1 . . . photographic lens

The invention claimed is:
 1. An image processing device, comprising: animage generation unit that is configured to generate an image at anoptionally selected focal plane of a subject from output data of aplurality of photodetectors disposed at each of a plurality ofmicrolenses; a correction unit that is configured to correct output dataof one of the plurality of photodetectors on which light from a firstarea of the subject and light from a second area of the subject areincident, based upon a point spread function that expresses an opticalspread of a light flux incident to the microlenses; and a control unitthat is configured to control the image generation unit so as togenerate pixels of the image each corresponding to the first area andthe second area.
 2. The image processing device according to claim 1,wherein: a number of the pixels which is generated by the imagegeneration unit is greater than a number of the microlenses.
 3. Theimage processing device according to claim 2, wherein: the correctionunit comprises; a Fourier transforming unit that is configured toperform a Fourier transformation upon the image generated by the imagegeneration unit; a calculation unit that is configured to effectivelydivide results of the Fourier transformation performed by the Fouriertransforming unit, by a Fourier image of the point spread function; andan inverse Fourier transforming unit that is configured to perform aninverse Fourier transformation upon results of the division by thecalculation unit.
 4. The image processing device according to claim 1,wherein: the correction unit comprises; a Fourier transforming unit thatis configured to perform a Fourier transformation upon the imagegenerated by the image generation unit; a calculation unit that isconfigured to effectively divide results of the Fourier transformationperformed by the Fourier transforming unit, by a Fourier image of thepoint spread function; and an inverse Fourier transforming unit that isconfigured to perform an inverse Fourier transformation upon results ofthe division by the calculation unit.
 5. An image-capturing devicecomprising: the image processing device according to claim 1.